r 

/  A0-A122  458  GRAPHICS  DISPLAY  TEST  VOLUME  1  TEST  PLAN(U)  MARINE 
X  CORPS  TACTICAL  SYSTEMS  SUPPORT  ACTIVITY  CAMP  PENDLETON 

CA,  R  P  ISBELL  ET  AL.  01  FEB  82  MCTSSA-22T001/U-TP-01 
UNCLASSIFIED  F/G  14/2  ^  NL 


A 1 22  458 


UNCLASSIFIED _ 

SECURITY  CLASSIFICATION  of  This  pace  Dote  Enter'd) 


REPORT  DOCUMENTATION  PAGE 

'  REPORT  NUMBER  J\l.  OOVT  ACCESSION  NO. 

JTSSA  DOCUMENT  NO.  22T001/U-TP-01 


TITLE  (and  Subtitle) 

1APHICS  DISPLAY  TEST 
3LUME  1  -  TEST  PLAN 


authors; 

aj  R.P.  Isbell  USMC 
apt  C.D.  Stephens  USMC 

■  PERFORMING  organization  name  ANO  AOORESS 

larine  Corps  Tactical  Systems  Support  Activity 
larine  Corps  Base 

lamp  Pendleton,  CA  920SS  _ 


I.  CONTROLLING  OFFICE  NAME  ANO  AOORESS 

Marine  Corps  Development  and  Education  Command 
^uantico,  VA  22134 

I*  MONITORING  AGENCY  NAME  A  AOORESSf/f  dltlorent  Irom  Controlling  Of(lco) 

Marine  Corps  Tactical  Systems  Support  Activity 

Marine  Corps  Base 

Camp  Pendleton,  CA  920SS 


l«.  OlSTRiauTlON  STATEMENT  ( a I  title  Report) 

Approved  for  Public  Release;  distribution  unlimited 


READ  INSTRUCTIONS 

_ BEFORE  COMPLETING  FORM 

3.  RECIPIENT’S  CATALOG  NUMBER 


S.  TYPE  OF  REPORT  ft  PERIOD  COVERED 

TEST  PLAN 

t.  PERFORMING  ORG.  REPORT  NUMBER 
i.  contract  or  grant  numbers 


10.  PROGRAM  ELEMENT.  PROJECT.  TASK 
AREA  ft  WORK  UNIT  NUMBERS 


12.  REPORT  OATE 

1  Feb  1982 


!3.  NUMBER  OF  PAGES 

41 


IS.  SECURITY  CLASS,  (of  thlt  report; 

UNCLASSIFIED 


I  So.  DECLASSIFICATION/  DOWNGRADING 
SCHEDULE 


17.  DISTRIBUTION  STATEMENT  (at  the  efterrecr  entered  In  Block  10,  It  dltlaranl  tram  Report) 

Approved  for  Public  Release;  distribution  unlimited 


IB.  supplementary  notes 

SEE  VOLUME  2 


DTIC 


If.  KEY  WO  nOS  fConrinu*  on  ravmeao  at  da  It  nacaaaory  ond  Identity  by  block  number) 

GRAPHIC  DISPLAY 

COMBAT  OPERATIONS  CENTER 

MILITARY  MAP  BACKGROUND 

USER  COMPATIBILITY 

m,:  _  _ 

TEST  PLAN 

TACTICAL 

DISPLAY  SIZE 

STATISTICAL  ANALYSIS 

E 

20.  ABSTRACT  (Continue  on  referee  tide  It  neceeeerr  end  Identity  by  block  number) 

Display  size  and  the  need  for  a  map  background  have  been  driving  factors  in  the 
size  and  weight  of  developing  automated  command  and  control  systems.  This  test 
plan  describes  a  test  designed  to  assess  the  effects  of  display  size  and  the 
presence/absence  of  a  military  map  background  as  they  relate  to  the  functional 
utility  of  the  graphics  display  of  an  envisioned  automated  command  and  control 
system.  Test  objectives,  measures  of  performance,  procedures,  and  analysis  are 
detailed... 


DO  I  JAN  71  1473  EDITION  OF  I  NOV  Bt  It  OBSOLETE 

$  N  0102- LF-OU- 660 1 


_ UNCLASSIFIED _ 

SECURITY  CL  Aftl  FI  CATION  OF  THIt  PAGE  (When  Dote  Eri«»r«r) 


MCTSSA  DOCUMENT  NO- 
22T00  1  /U-TP-0 1 


GRAPHICS  DISPLAY  TEST 


VOLUME  I 


TEST  PLAN 


MCTSSA 

MARINE  CORPS  BASE 
CAMP  PENDLETON,  CA  92055 


82  12  IS  055 


1  FEBRUARY  1982 


;»w«i 


C  r  a  c  h  i  a  s  risTln.’ 

Volume  I 
Te3t  Plan 


Prepared  by: 

Maj  3.  P.  ISBELL  USMC 
Capt  C.  D.  STEPHENS  UST-1C 


Reviewed  by: 

Maj  T.  BUSCSM I  .JR.  U?*!C 
Maj  D.  P.  ARTOTTS  USMC 


ABSTRACT:  Tisplay  size  and  the  need  for  a  map  background  have 
been  driving  factors  in  the  size  and  weight  of 
developing  automated  command  and  control  svstens.  This 
test  plan  describes  a  test  designed  to  assess  the 
effects  of  display  size  and  the  presence/absence  of  a 
military  map  background  as  they  relate  to  the 
functional  utility  of  the  graphics  display  of  an 
envisioned  automated  command  and  oontrol  system.  Test 
objectives,  measures  of  performance,  procedures,  and 
analysis  are  detailed. 


This  Test  Plan  is  a  working  document  and  does  not  represent 
official  policy  or  doctrine  of  the  United  States  Marine  Corps. 
The  contents  of  this  document  may  not  be  used  for  advertising 
purposes  and  should  not  be  considered  an  endorsement  of  any 
system. 


1  February  19S2 


Analysis  Section 

Tactical  Svstsms  Tevelopnent  Branch 
Marine  Corps  Tactical  Systems  Support  Activity 
Came  Pendleton,  California 


Accession  For 

NTIS  GRA&I 
DTIC  TAB 

Uuanncunoed 
Justif a  cot ' , n 


By- 


Distribution/  I 

Availability  Codes  j 

Avail  and/or  j 

Dist  |  Special 


TABLE  OF  CONTENTS 


Section/Paragraph  Page 

1  INTRODUCTION  .  1-1 


1.1  Purpose  . i-i 

1.2  Test  Objectives  .  1-1 

1.2.1  Objective  1  1-1 

1.2.2  Objective  2  .  1 -1 

1.3  Measures  of  Performance  .  1-1 

1.3.1  OBJECTIVE  1  1-1 

1.3. 1.1  NOP  la  .  1-1 

1.3. 1.2  MOP  1b  .  1-1 

1.3.2  Objective  2  1-1 

1.3. 2.1  MOP  2a  .  1-1 

1.3. 2. 2  MOP  2b  .  1-2 

1.4  Factors  . 1-2 

1.4.1  Display  Site  . 1-2 

1.4.2  Map  Background  .  1-2 

1.4.3  Display  Types  .  1-2 

1.4.4  Other  Factors  .  1-2 

1.5  Scenario  .  1-3 

2  TEST  CONCEPT  .  2-1 

2.1  Approach  .  2-1 

2.2  Schedule  .  2-1 

2.2.1  Teat  Period  . . 2-1 

2.3  Personnel  Requires ents  . .  2-2 

2.3.1  Special  Qualifications  .  2-2 

2.4  Material  Requireaents  .  2-2 

2.4.1  Facilities  .  2-2 

2.4.2  Hardware  . .  2-2 

2.4.3  Software  . 2-3 

2.4.4  Data  Base  . 2-3 

2.5  Training  .  2-3 

2.5.1  CST  Personnel  . 2-3 

2.5.2  Test  Participants  . -  2-3 


i 


Section/Paragraoh  Page 

3  DATA  MANAGEMENT  . 


3.1  Data  Sources  . 

3*2  Data  Collection  in  Support  of  Objectives 

3.2.1  Objective  1  . . 

3. 2. 1.1  MOP  la  . 

3.2. 1.2  MOP  1b  . 

3.2.2  Objective  2  . . 

3.2. 2.1  MOP  2a  . 

3. 2.2. 2  MOP  2b  . 

3*3  Data  Reduction  . 

4  DATA  ANALYSIS  . 


3-1 

3-1 

3-1 

3-1 

3-1 

3-1 

3-1 

3-1 

3-1 

3- 1 

4- 1 


4.1  Objective  1  . . .  4-1 

4.1.1  Assumptions  . .  4-1 

4.1.2  Method  of  Analysis  .  4-1 

4.1.3  Interpretation  of  Results  . 4-2 

4.2  Objective  2  .  4-2 

4.2.1  Method  of  Analysis  .  4-2 

4.2.2  Interpretation  of  Results  .  4-4 


APPENDIX  A  Test  Scenario  .  A-1 

APPENDIX  B  Data  Manage* ent  System  .  B-1 

APPENDIX  C  Detailed  Analysis  . .  C-1 


ii 


SECTION  1 
INTRODUCTION 


1.1  PURPOSE.  The  purpose  of  the  Graphics  Display  Test  is  to  assess  the 
effects  of  display  size  and  the  presence/ absence  of  a  nap  background  as  they 
relate  to  the  functional  utility  of  the  graphics  display  for  the  battalion 
level  Tactical  Combat  Operations  (TCO)  supported  Combat  Operations  Center 
(COC). 


1.2  TEST  OBJECTIVE 

1.2.1  Objective  1 .  Objective  1  is  to  evaluate  the  effectiveness  of  the 
various  types  of  graphics  displays  in  providing  assistance  to  the  commander. 

1.2.2  Objective  2.  Objective  2  is  to  assess  the  user  compatibility  of  the 
various  types  of  graphics  displays. 

1.3  MEASURES  OF  PERFORMANCE 

1.3.1  Objective  1 .  Two  measures  of  performance  (MOP)  will  be  used  for 
Objective  1 . 

1.3. 1.1  MOP  la.  The  first  MOP  for  Objective  1  will  be  the  cumulative  tine 
required  to  extract  several  pieces  of  tactical  information,  i.e.,  information 
of  the  sort  that  is  necessary  in  formulating  tactical  plans  and  making 
tactical  decisions,  from  ths  graphics  display. 

1.3. 1.2  MOP  1b.  The  second  MOP  for  Objective  1  will  be  the  total  score 
received  by  the  test  participant  on  the  answers  given  to  several  tactical 
questions. 

1.3.2  Objective  2.  Two  HOPs  will  be  used  for  Objective  2. 

1.3. 2.1  MOP  2a.  The  first  MOP  for  Objective  2  will  be  ordinal  Judffrents 
expressed  by  each  of  the  test  participants. 
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1.3. 2. 2  MOP  2b.  The  second  MOP  for  Objective  2  will  be  qualitative 
assessments  by  each  participant  ss  to  the  compatibility  of  the  various 

displays. 

1 .4  FACTORS.  The  two  principal  factors  which  will  be  tested  are  display  size 
and  presence/ absence  of  a  map  background .  The  test  will  investigate  the 
interaction  effect  of  the  two  principal  factors. 

1.4.t  Display  Site.  Two  levels  of  display  size  will  be  utilized.  The  large 
size  will  be  10-3/4"  by  10-3/4"  [the  approximate  size  of  the  Dynamic  Situation 
Display  (DSD)  as  described  in  the  TOO  System  Description  Document  (SDD)].  The 
small  size  will  be  approximately  4-1/2"  by  4-1/2". 

1.4.2  Map  Background.  Two  levels  of  map  background  will  be  utilized.  The 
first  level  will  be  the  presence  of  a  map  background  representing  the  detail 
of  a  1:50,000  paper  map  behind  the  tactical  symbology  on  the  graphics  display. 
The  second  level  will  be  a  background  consisting  of  only  gridlines  behind  the 
tactical  symbology.  In  the  latter  case  the  test  participants  will  be  provided 
with  a  paper  map. 

1.4.3  Display  Types.  Combinations  of  the  two  levels  of  each  of  the  two 
principal  factors  will  be  utilized  so  that  four  display  types  are  tested.  The 
four  types  are: 

a.  Large  display  with  map  background  (Type  A) 

b.  Large  display  without  map  background  (Type  B) 

c.  Small  display  with  map  background  (Type  C) 

b.  Small  display  without  map  background  (Type  D) 

1.4.4  Other  Pactora 


Other  factors  such  as  differences  between  test  participants  and 
scenarios,  learning  factor  and  experimentation  order  will  not  be  explicitly 
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tested.  These  factors  will  be  controlled  through  a  relatively  large  sample 
(<13  test  iterations  are  planned),  efforts  to  minimize  variance  among 
participants  and  scenarios,  and  deliberate  confounding. 

Test  participants  will  be  24  officers  holding  primary  ground  combat  30Sa 
or  other  officers  with  recent  tactical  training  such  as  Amphibious  Warfare 
School  (AVS).  Bach  of  these  participants  will  work  with  two  of  the  four 
possible  display  types.  The  types  of  displays  used  by  the  participants  and 
the  order  in  which  they  are  used  will  be  varied  in  accordance  with  recognised 
experimental  design  procedures. 

1.5  SCENARIO.  Two  scenarios  will  be  used,  both  of  which  will  involve  a 
reinforced  Marine  infantry  battalion  conducting  independent  operations  within 
a  tactical  area  of  responsibility  (TAOS). 
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SECTION  2 


TEST  CONCEPT 

2.1  APPROACH.  The  Graphics  Display  Test  is  designed  to  produce  a  represen¬ 
tative  coabet  scenario  at  the  infantry  battalion  level  in  order  to  evaluate 
the  effectiveness  of  the  various  graphics  displays  in  providing  necessary 
information  to  the  test  participants.  The  baseline  graphics  display  will  be  a 
large  display  with  a  nap  background  (Type  A).  The  other  display  types  are 
alternative  to  this  baseline. 

2.2  SCHEDULE.  The  test  will  be  prepared  and  conducted  in  accordance  with  the 
following  schedule: 

a.  Test  Preparation  —  16  Nove^Tier  1981  to  11  January  1982 

b.  Pilot  Test  —  11  to  15  January  1982 

c.  Teat  Refinements  —  15  to  30  January  1982. 

d.  Teat  Period  —  1  to  1 1  Pebruary  1982. 

2.2.1  Teat  Period.  The  teat  period  will  run  the  first  two  weeks  of  Pebruary 
1982,  in  accordance  with  the  following  dally  schedule: 


Number  of  Displays 


Day 

Event 

Particioants 

Tested 

Non  1 

Peb  Pinal  GTP  Preparation 

AH 

PH 

CM 

• 

£ 

Peb 

Test 

4  participants 

AB,3C 

CD,  BA 

Wed  3 

Peb 

Test 

4  participants 

CB.DC 

AC.BD 

Thu  4 

Peb 

Test 

4  participants 

CA.AD 

DA,DB 

Fri  5 

Peb  GTP 

Maintenance 

Hon  8 

Peb  GTP 

Maintenance 

Tue  9 

Peb 

Test 

4  participants 

CD,  BA 

AB.BC 

Wed  10  Peb 

Test 

a  participants 

AD,  3D 

CB.DC 

Thu  1 1 

Peb 

Test 

4  participants 

DA,DB 

CA.AD 

Aa  indicated  above,  four  participants  will  be  utilized  on  each  teat  day.  Each 
day  will  be  divided  into  an  AM  and  PM  session  with  two  participants  involved 
in  each  session.  Each  individual  participant  will  be  tested  on  two  of  the 
four  possible  displays  in  accordance  with  the  schedule  given  above,  e.g.,  on 
the  first  test  day  the  first  participant  in  the  AM  session  will  be  tested  on 
display  A  then  on  display  B,  etc. 

2.3  PERSONNEL  REQUIREMENTS.  Personnel  requirements  for  the  Graphics  Display 
Test  will  be  provided  as  follows: 

a.  Test  preparation  will  be  accomplished  by  Marine  Corps  and  on-site 
contractor  personnel  at  MCTSSA. 

b.  Pilot  Test  participants  will  be  a  limited  number  of  officers  from 
MCTSSA  who  are  not  involved  in  test  design  or  preparation. 

c.  Test  participants  will  be  Marine  officers  provided  by  LFTC  Pac  and  1st 
Marine  Division. 

d.  The  Control  Simulation  Team  (CST)  will  consist  of  Marine  Corps  and 
on-site  contractor  personnel  from  MCTSSA. 

e.  The  Data  Management  Team  (DMT)  will  be  Marine  Corps  officers  assigned 
to  the  Analysis  Section  augmented  by  enlisted  personnel  from  MCTSSA. 

2*3.1  Special  Qualifications.  Test  participants  will  be  Marine  officers 
holding  a  primary  ground  combat  MOS  (03,  08,  18)  or  other  officers  with  recent 
tactical  training  such  as  AVS. 

2.4  MATERIAL  REQUIREMENTS 

2*4.1  Facilities.  The  test  facilities  required  for  this  test  are  the  MCTSSA 
Generalised  Test  Facility  (GTF)  located  in  building  31330  and  the  MCTSSA 
computer  facility  located  in  building  31337. 

2.4.2  Hardware.  Hardware  for  the  test  will  consist  of  the  GTF  suite  of 
equipment  and  the  PD?  i1/**0  system. 
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2.4.3  Software.  Software  for  the  teat  will  consist  of  the  GTF  system 
program,  DECNET,  IAS  and  3SX-11M. 

2.4.4  Data  3ase.  Data  Base  Management  Systea-11  (DBMS-11)  will  be  used  for 
the  teat. 

2.5  TRAI5IHG 

2.5.1  GST  Personnel.  Members  of  the  CST  will  be  briefed  on  test  objectives, 
teat  duties  and  test  requirements  and  will  participate  in  test  rehearsals  and 
a  pilot  test  prior  to  the  scheduled  test  period. 

2.5.2  Test  Participants.  Training  for  test  participants  will  consist  of  a 
pretest  briefing  covering  test  objectives,  test  approach  and  participant 
duties.  The  briefing  will  be  followed  by  "hands  on"  instructions  on  the  GTF 
equipment  that  will  be  utilized  by  the  participants. 


SECTION  3 
DATA  MAHAGEMEN' 


3.1  DATA  SOURCES.  Data  will  be  collected  for  analysis  and  evaluation  to 
support  the  test  objectives.  Data  will  be  collected  from  test  participants  by 
a  combination  of  verbal  response  to  test  observers  and  written  response  to 
questionnaires . 

3.2  DATA  COLLECTION  IN  SUPPORT  0?  OBJECTIVES 

3.2.1  Objective  1 

3. 2. 1.1  MOP  1  a.  The  data  for  this  MOP  will  be  the  total  time  (meaaured  in 
seconds)  that  the  teat  participant  uses  in  extracting  (from  the  graphics 
display)  the  information  required  to  respond  to  six  tactical  questions. 

3.2. 1.2  MOP  1b.  The  data  for  this  MOP  will  be  a  written  record  of  the  test 
participants'  responses  to  a  series  of  six  tactical  questions. 

3«2.2  Objective  2 

3*2.2. 1  MOP  2a.  The  data  for  this  MOP  will  be  a  written  record  of  the  test 
particlpanta'  ordinal  rankings  of  the  four  displays  as  a  function  of  the 
degree  of  comfort  they  felt  with  the  displays.  Por  the  purposes  of  this  test 
the  word  "comfort"  implies  attributes  such  aa  "ease  of  use",  "selectivity", 
"flexibility",  and  "adaptability"  (see  HCTSSA  Doc.  No.  24T001 /U-TN-08, 

Handbook  for  TCO  Test  and  Evaluation).  Each  participant  will  be  tested  on 
only  two  of  the  display  types  but  will  be  familiarized  with  all  four  prior  to 
collecting  data  for  this  MOP. 

3* 2. 2. 2  MOP  2b.  The  data  for  this  MOP  will  be  the  recorded  comments  of  each 
participant  concerning  his  assessment  of  the  compatibility  of  the  various 
displays . 

3*3  DATA  REDUCTION.  Data  reduction  will  take  place  at  the  conclusion  of  each 
day  of  testing  and  will  be  recorded  on  the  data  reduction  fonts  contained  in 
Appendix  B. 
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SECTION  4 
DATA  ANALYSIS 


4. 1  OBJECTIVE  1 

4.1.1  Aaaunrotiona.  It  ia  assumed  that  the  various  measures  are  independent 
and  normally  distributed  random  variables,  that  all  observations  of  a  given 
MOP  have  the  same  variance  and  that  the  mean  of  each  measure  is  a  linear 
combination  of  certain  unknown  parameters. 

4.1.2  Method  of  Analysis.  Data  analysis  for  Objective  1  will  be  based  on  a 
22  factorial  design  (see  Cochran  and  Cox,  "Experimental  Design",  Wiley  1957). 
The  factors  and  levels  of  factors  are  as  specified  in  paragraph  1.4. 

The  null  hypotheses  for  the  test  are  that: 

a.  There  is  not  a  statistically  significant  difference  between  the  large 
display  and  the  small  display  as  measured  by  the  stated  MOPs  for  Objective  1 . 

b.  There  is  not  a  statistically  significant  difference  between  a  display 
with  and  a  display  without  a  map  background  as  measured  by  the  stated  MOPs  for 
Objective  1 . 

c.  There  is  not  a  statistically  significant  interaction  effect  between 
the  two  principal  factors  as  measured  by  the  MOPs  for  objective  1 . 

The  analysis  of  variance  (ANOVA)  used  to  test  the  above  hypotheses  will 
be  based  on  48  observations  (see  Appendix  C  for  a  detailed  mathematical 
methodology) .  Table  4-1  gives  a  breakdown  of  the  48  observations  where  the 
display  types  are  as  given  in  paragraph  1.4.3.  This  design  allows  for  12 
observations  of  each  display  type. 

Each  of  the  24  test  participants  will  be  randomly  assigned  to  two  display 
types  such  that  four  participants  will  utilize  each  of  the  six  combinations  of 
types  (four  types  taken  two  at  a  time).  Within  these  subgroups,  the  order  in 
which  display  types  are  utilized  will  be  equally  divided.  Since  there  will  be 
a  morning  and  afternoon  session,  an  equal  number  of  identical  pairs  of 
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displays  will  be  tested  in  the  A?1  and  PM  (see  schedule  in  paragraph  2.2.1). 
Finally,  the  two  scenarios  will  be  used  an  equal  number  of  times  with  each 
display  type.  This  will  be  accomplished  by  having  each  test  participant  use 
Scenario  I  on  his  first  iteration  and  Scenario  II  on  his  second  iteration 
during  the  first  week  of  the  test  and  reversing  the  order  during  the  second 
week  of  the  test. 

4.1.3  Interpretation  of  Results.  After  the  AHOVAs  for  MOP  la  and  MOP  1b  have 
been  performed  a  P-value  for  each  null  hypothesis  will  be  calculated.  A 
P-value  is  a  statement  of  the  probability  that,  if  the  null  hypothesis  is 
rejected,  you  are  in  fact  rejecting  the  true  case. 

It  is  common  that  the  null  hypothesis  is  not  rejected  unless  the  P-value  is 
less  than  a  prescribed  significance  level  (denoted  ).  Typical  significance 
levels  are  »  .05  or  *0.1. 

4.2  OBJECTIVE  2 


4.2.1  Method  of  Analysis 


Although  subjective  in  nature,  MOP  2a  will  be  assessed  by  constructing  an 
interval  scale  based  on  the  test  participants'  ordinal  rankings  of  the  four 
display  types  (Glenn  F.  Lindsay,  "On  Constructing  Interval  Scales  from  Ordinal 
Judgements",  1977,  unpublished).  As  a  check  on  the  validity  of  the  test 
participants '  ordinal  rankings,  the  Kendall  coefficient  of  concordance  will  be 
used  to  test  the  hypothesis  that  the  participants'  judgments  are  unrelated/- 
inconsistent  (see  Siegel,  "Sonparametric  Statistics",  McGraw-Hill,  1956).  A 
detailed  methodology  for  both  procedures  is  given  in  Appendix  C. 


The  subjective  assessments  submitted  as  MOP  2b  will  be  quantitatively 
analysed  only  to  the  extent  that  percentages  of  participants  expressing  a 
common  observation  will  be  identified.  Otherwise  a  general  summary  of  the 
assessments  will  be  presented  and  commented  on  in  the  test  report. 
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TA3LE  4-1 

3REAKD0VN  OF  TEST  OBSERVATIONS 


DISPLAY  TYPE 

NUMBER  OP 

NUMBER 

OF 

COMBINATIONS 

PARTICIPANTS 

ORDER 

OBSERVATIONS 

2 

A 

then 

B 

4 

AB 

4 

< 

> 

8 

2 

B 

then 

A 

4 

2 

A 

then 

W 

4 

AC 

4 

< 

> 

8 

2 

C 

then 

A 

4 

2 

A 

then 

D 

4 

AD 

4 

< 

> 

8 

2 

D 

then 

A 

4 

2 

B 

then 

C 

BC 

4 

< 

8 

2 

C 

then 

B 

2 

B 

then 

D 

BD 

4 

< 

8 

2 

D 

then 

B 

2 

C 

then 

D 

4 

CD 

4 

< 

> 

8 

2 

D 

then 

C 

4 

Totals 

24 

48 
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4.2.2  Interpretation  of  Results 

The  interpretation  that  can  be  made  of  an  interval  scale  is  best  explained 
by  an  example.  If  four  instances  of  some  property  being  measured  are  denoted 
by  the  letters  E,  F,  G  and  H  (for  this  test  the  property  being  measured  is 
"user  compatibility"  and  the  instances  are  the  four  display  types)  then  an 
interval  measurement  of  the  degree  of  the  property  possessed  by  the  instances 
might  be  presented  by, 

__  i  __  i _ . .  , _  _ _ _ *  — » ___________ 

G  F  EH 

On  such  a  scale,  the  base  or  zero  point  as  veil  as  the  units  are  entirely 
arbitrary.  It  could  be  concluded  that  E  and  H  possess  more  of  the  property 
than  G  and  F,  that  there  appears  to  be  little  difference  between  E  and  H  as 
compared  with  the  difference  between  E  and  F,  etc.  If  the  intervals  between 
all  instances  were  approximately  equal  then  very  little  information  is 
provided.  In  such  a  case,  since  the  units  are  arbitrary,  the  intervals  could 
represent  large  but  equal  differences  or  small  but  equal  differences. 

A  high  or  significant  Kendall  coefficient  of  concordance  may  be 
interpreted  as  meaning  that  the  test  participants  are  applying  essentially  the 
same  standard  in  ranking  the  display.  If  such  is  the  case,  confidence  is 
added  to  the  calculated  interval  scale. 


APPENDIX  A 


SCENARIOS 

A.1  INTRODUCTION.  There  are  two  scenarios,  both  of  which  involve  a  rein¬ 
forced  Narine  infantry  battalion  conducting  independent  operations  within  a 
tactical  area  of  responsibility  (TAOR). 

A. 2  SCENARIO  I 

On  D-day  the  Third  Narine  Anphibioua  Force  (ill  MAP)  conducted  an 
aaphibious  assault  on  the  beaches  north  of  Oceanside,  California.  An  enemy 
motorized  rifle  regiment  is  located  north  of  Oceanside  at  the  Santa  Margarita 
Airfield.  This  airfield  is  a  NAP  objective.  An  enemy  motorized  rifle 
battalion  is  located  in  the  city  of  Escondido,  east  of  Oceanside.  The  NAP 
commander  considers  it  essential  that  the  enemy  be  denied  the  now  deserted 
Palos*  Airfield  to  the  south  of  Oceanside.  Accordingly,  he  specified  that 
1st  Battalion,  4th  Narines  (reinforced  with  TONS,  engineers  and  sensor  assets) 
be  helicopter  lifted  to  the  Palomar  Airfield  on  D-day  with  the  mission  to 
seise  and  defend  the  airfield,  denying  its  use  to  the  enemy  until  relieved  by 
follow-on  forces.  The  NAP  commander  designated  a  TAOR  for  this  purpose  and 
initially  placed  no  restrictions  upon  fire  or  maneuver  by  the  battalion 
commander  outside  the  TAOR.  Upon  order,  however,  the  battalion  commander  will 
be  restricted  to  fire  and  maneuver  within  the  TAOR  as  friendly  forces  approach 
the  airfield. 

It  is  now  0*1 ,  III  NAP  is  ashore  and  the  Santa  Margarita  Airfield  has  been 
taken.  The  1st  Battalion,  4th  Narines  is  situated  within  its  TAOR  at  the 
Palomar  Airfield  and  has  experienced  no  enemy  contact  up  to  now.  The 
battalion  commander  has  designated  company  areas  of  responsibility  and 
instituted  vigorous,  continuous  patrolling  inside  and  outside  of  the  TAOR. 

A. 3  SCENARIO  II.  The  Third  Narine  Division  is  garrisoned  in  the  city  of 
Escondido,  California.  An  enemy  amphibious  assault  is  probable.  The  assault 
nay  occur  on  the  beaches  north  of  the  city  of  Oceanside,  the  beach  south  of 
the  city  of  Carlsbad,  or  on  the  beach  at  Del  Mar  located  south* of  the  town  of 
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Solans  3each.  Front  its  location  at  Escondido,  the  Third  Division  is  in  a  goo d 
position  to  react  to  any  of  the  possible  landing  sites.  There  are,  however, 
two  radio  towers  located  northwest  of  the  city  of  Vista  which  are  of 
potentially  critical  political  significance.  In  order  to  deny  these  towers  to 
the  enemy  the  division  commander  has  assigned  to  the  1st  Sattalion,  4th 
Narines  (reinforced  with  TOVS,  engineers  and  sensor  assets)  a  TAOR  surrounding 
those  towers,  initially  placing  no  restrictions  upon  the  fire  and  maneuver  by 
the  battalion  comander  outside  the  TAOR.  Upon  order,  the  battalion  may  be 
restricted  to  fire  and  maneuver  within  the  TOAR.  This  will  occur  if  other 
friendly  forces  are  deployed  into  that  area.  The  battalion  commander  has 
designated  company  areas  of  responsibility  and  has  instituted  vigorous, 
continuous  patrolling  inside  and  outside  of  the  TAOR.  The  enemy  amphibious 
aasault  is  now  imminent. 
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APPENDIX  B 

DATA  MANAGEMENT  SYSTEM 

B.1  INTRODUCTION.  The  purpose  of  the  data  management  system  is  to  organise 
the  data  for  the  test  to  ensure  that  all  required  data  is  collected  and 
handled  in  an  orderly  manner. 

B.2  DATA  ORGANIZATION.  A  separate  data  file  will  be  kept  for  each  test 
participant  $  the  participants  will  be  designated  as  Test  Participant  #1  thru 
Test  Participant  #24.  Each  file  will  contain  the  following  forms: 


a.  Data  File  Cover  Information  (Tab  1 ) 

b.  Participant  Profile  Form  (Tab  2) 

c.  MOP  la  Questionnaire  for  Scenario  I  (Tab  3) 

d.  MOP  la  Questionnaire  for  Scenario  II  (Tab  4) 

e.  MOP  la  Data  Collection  Form  (Tab  5) 

f.  MOP  lb  Questionnaire  for  Scenario  I  (Tab  6) 

g.  MOP  lb  Questionnaire  for  Scenario  II  (Tab  7) 

h.  MOP  2a  Instructions/Data  Collection  Form  (Tab  S) 

1.  MOP  2b  Questionnaire/ Comment  Sheet  (Tab  9) 

j.  MOP  lb  Score  Sheet  for  Scenario  I  (Tab  10) 

k.  MOP  1b  Score  Sheet  for  Scenario  II  (Tab  11 ) 


3.3  DATA  MANAGEMENT .  Data  Management  for  the  test  will  consist  of  two  major 
functions,  data  collection  and  data  reduction. 

B.3.1  Data  Collection.  At  the  beginning  of  each  test  day  the  data  manager 
will  fill  out  the  cover  information  on  each  data  file  and  deliver  the 
appropriate  data  files  to  the  work  station  test  controllers.  The  work  station 
test  controllers  will  have  the  participant  complete  the  Participant  Profile 
Form  and  ensure  that  all  data  is  collected  in  accordance  with  the  following 
paragraphs. 

B.3.1.1  MOP  la.  The  work  station  test  controller  will  ask  the  test  partici¬ 
pant  to  face  away  from  the  graphics  display  and  then  present  him  with  the  MOP 
la  questionnaire  corresponding  to  the  test  scenario  being  utilised.  The  test 
participant  will  be  allowed  to  become  familiar  with  the  questions-  and  to 
request  clarification  if  necessary.  After  the  participant  indicates  that  he 
is  ready  to  proceed,  the  work  station  test  contoller  will  have  the  participant 
face  the  display  and  will  start  the  timer.  After  the  participant  has  recorded 


his  response  to  all  questions  the  work  station  test  controller  will  stop  the 
timer.  The  work  station  test  controller  will  then  record  the  total  elapsed 
time  (to  the  nearest  second)  on  the  MOP  la  Bata  Collection  Fora  and  return  all 
forms  to  the  data  file. 

(WOTE:  The  teat  participant  will  be  asked  six  questions  which  require  infor¬ 
mation  from  the  graphics  display.  Two  questions  will  require  use  of  overlay 
information  only,  two  will  require  map  information  only  and  two  will  require 
information  from  both  the  map  and  the  overlay.) 

3. 3. 1.2  WOP  1b.  The  work  station  test  controller  will  present  the  partici¬ 
pant  with  the  WOP  1b  questionnaire  corresponding  to  the  scenario  being  uti¬ 
lised  and  ask  him  to  provide  a  response  to  each  of  the  six  tactical  questions. 

After  the  participant  has  completed  the  questionnaire  the  work  station  test 
controller  will  enter  the  information  called  for  on  the  form  and  return  it  to 
the  data  file. 

B.3.I.3  WOP  2a.  After  the  participant  has  completed  the  teat  on  the  two 
displays  assigned  to  him  sad  has  had  an  adaquata  amount  of  time  to  become 
familiar  with  the  two  diaplaya  not  utilized,  the  work  station  test  controller 
will  explain  the  information  required  for  WOP  2a.  The  teat  controller  will 
then  present-  the  participant  with  four  cards  labeled  A,  8,  C  and  0  and  have 
the  participant  arrange  the  cards  in  accordance  with  the  instructions  given. 
The  teat  controller  will  than  transfer  the  Information  to  the  WOP  2a 
Instruction/Data  Collection  Pom  and  return  all  forms  to  the  data  fils. 

3.3. 1 .4  WOP  2h.  At  the  conclusion  of  the  teat  the  work  station  teat  con¬ 
troller  will  have  the  participant  complete  the  WOP  2b  Questionnaire/Comment 
Sheet  and  then  return  it  to  the  data  file. 

3. 3.2  Data  Reduction 

B.3.2. 1  Daily.  After  each  participant  has  completed  both  test  iterations, 
the  work  atation  test  controller  will  transfer  the  participant's  responses  to 
the  MOP  1b  questionnaire  to  the  appropriate  WOP  1b  score  sheets,  entering  the 
total  score  in  the  space  provided  on  the  fora.  The  work  station  teat  con¬ 
troller  will  than  ensure  that  all  other  forma  and  questionnaires  are  complete, 
place  all  forms/ questionnaires  in  the  data  file  and  return  the  data  file  to 
the  data  manager. 

B.3.2. 2  Data  Consolidation.  At  the  conclusion  of  the  taat  period  the  data 
manager  will  consolidate  all  data  utilizing  the  following  forma: 


a.  WOP  la  Data  Consolidation  Form  (Tab  12) 

b.  WOP  1b  Data  Consolidation  Form  (Tab  13) 

c.  WOP  2a  Data  Consolidation  Pom  (Tab  14) 

d.  Data  Consolidation  Fora  for  Kendall  Coaffieiant  (Tab  19) 


TAB  1 


Participant's 


Participant  # 


Iteration  #1 : 


Iteration  #2: 


DATA  FILS  COVES  INFORMATION 


Nane 

Clast)  (first)  (m.i.y 


AH/PM  Session 


Display  Type  Scenario 


Display  Type  __________  Scenario 
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TEST  PARTICIPANT  PROFILE 


NAME: 


RANK: 


YEARS  OP  SERVICE 


MOSs  (0 

(2) 

(3) 

RELATED 

/ 

/ 

/ 

BILLET/ 

TIME 

/ 

/ 

/ 

SERVED 

/ 

/ 

/ 

/ 

/ 

/ 

• 

/ 

/ 

• 

/ 

/ 

/ 

/ 

/ 

/ 

/ 

AVS? 

COMMAND  4  STAPP? 

OTHER: 

Y®a  /  No 

Y«s  /  No 

MOP  la  Q’JESTIONfTAIRS 
(SCENARIO  I) 


Participant  # 


1 •  Which  patrol  is  closest  to  the  water  tower  located  approximately  700m  west 
of  the  TAOR? 


Answer: 

2.  Which  company  CP  is  farthest  away  from  the  battalion  CP? 

Answer: 

3*  What  are  the  six  digit  grid  coordinates  of. the  water  tower  located  west  of 
the  TAOR? 


Answer: 


4.  Suppose  TOWS  are  located  at  751  668;  their  range  is  3000m.  Considering 
terrain,  can  those  TOWS  hit  a  target  located  at  the  intersection  at  734  672? 


Answer: 


5.  Which  patrol  is  closest  to  the  battalion  OP? 


Answer: 


6.  What  is  the  distance  between  the  battalion  OP  and  the  water  tower  located 
west  of  the  TAOR? 


Answer: 


I 


TAB  4 

MOP  la  QUESTIONNAIRE 
(SCENARIO  II) 


Participant  # 


1 .  What  is  tha  distance  between  the  battalion  OP  and  tha  spot  south  of  the  OP 
vhara  the  hard  surfaced  road  crosses  the  railroad? 

Answer: 


2.  Suppose  TOWS  are  located  at  738  767;  their  range  is  3000m.  Considering 
terrain,  can  those  TOWS  hit  a  target  located  at  the  intersection  at  728  781? 

Answer: 


3.  Which  patrol  is  closest  to  the  battalion  OP? 
Answer: 


4.  Which  company  CP  is  farthest  away  from  the  battalion  CP? 
Answer: 


5.  What  is  the  six  digit  grid  coordinate  of  the  spot  south  of  the  battalion 
OP  where  the  hard  surfaced  road  crosses  the  railroad? 

Answer: 


6.  Which  company  CP  is  farther  from  Cuajome  Lalce? 
Answer: 


l 


TAB  5 


MOP  la  DATA 


Participant  # 


Scenario  I: 

_  Min  X  60  * 


COLLECTION 


Sec  * 


Scenario  II 


TAB  6 


'TCP  1b  QUESTIONNAIRE 
(SCENARIO  I) 


Participant  #  ___________ 

1.  What  is  the  moat  likely  spot  for  an  LZ  for  an  enemy  helicopter  borne 
assault  on  the  Palomar  Airfield? 

a.  TIC  715  674 

b.  VIC  740  655 

c.  VIC  760  655 

d.  VIC  756  649 

2.  What  is  the  most  likely  enemy  avenue  of  approach  for  a  nonmechanized  enemy 
attack  from  the  south? 

a.  Along  the  axis  715  635  to  725  645  to  the  airfield. 

b.  Up  the  hard  surfaced  road  from  750  632  to  749  640  to  752  650  to  the 
airfield. 

c.  Up  the  draw  from  756  640  to  756  646  to  754  650  to  the  airfield. 

d.  Prom  773  630  to  733  640  to  the  airfield. 

3>  The  battalion  commander  is  considering  constituting  a  platoon-sized 
reserve.  What  would  you  recommend? 

a.  Take  a  platoon  from  A  Co. 

b.  Take  a  platoon  from  B  Co. 

c .  Take  a  pi/  toon  from  C  Co . 

d.  Do  not  constitute  a  platoon  sized  reserve. 

4.  Reliable  intelligence  reports  a  platoon  of  enemy  tanks  advancing  towards 
the  airfield  from  the  east  along  Palomar  Airport  Road.  Which  is  the  best 
position  for  TOWS? 

a.  VIC  765  649 

b.  VIC  754  664 

c.  VIC  763  657 

d.  VIC  759  670 


-j  ' 


TAB  7 


MO?  1b  QUESTIONNAIRE 
(SCENARIO  II) 


Participant  #  _ 

1 .  What  is  tha  most  likely  spot  for  an  LZ  for  an  enemy  helicopter  borne 
assault  on  the  radio  towers? 

a.  VIC  745  772 

b.  VIC  717  778 

a.  VIC  735  777 

d.  VIC  752  753 

2.  What  is  the  most  likely  avenue  of  approach  for  a  nonraechani zed  enemy 

attack  on  the  radio  towers  from  the  west? 

a.  Along  Mission  Ave.  (Rte.  76)  from  722  776  to  739  784,  then  down  Santa 
Pe  Ave.  to  752  778  and  on  to  the  towers. 

b.  Along  the  railroad  tracks  from  737  740  to  752  750  then  north  to  the 
towers . 


c.  Prom  716  766  to  728  76 2  to  the  towers. 

d.  Prom  720  740  to  732  750  to  740  760  to  the  towers. 

3.  The  battalion  commander  is  considering  constituting  a  platoon-sized 
reserve.  What  would  you  reconend? 

a.  Taking  a  platoon  form  A  Company. 

b.  Taking  a  platoon  from  B  Company. 

c.  Taking  a  platoon  from  C  Company. 

d.  Not  constituting  a  platoon- si zed  reserve. 

4.  Reliable  intelligence  reports  a  platoon  of  enemy  tanks  advancing  towards 
the  TAOR  from  the  west  along  Mission  Avenue.  Which  is  the  best  location  for 
TOWS? 

a.  VIC  731  768 

b.  VIC  747  764 


c.  VIC  737  760 

d.  VIC  738  775 


5.  A  battalion-si zed  enemy  force  is  attacking  from  the  east  along  the  Palonar 
Airport  Road  axis.  Two  destroyers  have  been  placed  in  direct  support  of  1st 
Battalion,  d.th  Marines.  The  battalion  commander  has  designated  the  eastern 
boundary  of  the  TAOR  as  the  FEB A  and  deployed  A  Company  and  C  Company  in 
defensive  positions  along  the  7EBA  with  the  same  company  boundary.  He  has 
designated  3  Company  as  the  battalion  reserve.  Where  would  you  recommend 
placing  B  Company? 

a.  Do  not  move  them,  i.e.,  keep  them  at  VIC  3  Company  CP. 

b.  VIC  Battalion  CP 

c.  VIC  751  644 

d.  VIC  752  554 

6.  Continuing  the  situation  in  question  ?5,  where  would  you  recommend 
relocating  the  31  ms  mortar  platoon? 

a.  Do  not  move  them 

b.  VIC  752  656 

c.  VIC  722  669 


d.  VIC  740  650 


TAB  9 


MOP  2a  INSTRUCTIONS /DATA  COLLECTION  FORM 


Participant  # 


The  participant  will  rank  the  four  displays  in  the  order  that  he  feels 
that  they  possess  the  property  of  "user  compatibility",  i.e.,  in  the  order 
that  he  felt  comfortable  with  the  display.  The  participant  is  encouraged  to 
rank  all  displays  but  he  oay  celt  one  or  more  if  he  feels  he  can  not  judge 
that  display.  Ties  are  permitted  (write  ties  on  the  sane  line).  The 
participant  should  arrange  the  four  cards  marked  A  thru  D  until  he  is 
satisfied  with  the  order. 

Type  A  -  Large  Display  with  Map  Background 

Type  B  -  Large  Display  without  Map  Background 

Type  C  -  Snail  Display  with  Map  Background 

Type  D  -  Snail  Display  without  Map  Background 


Most  Conpatible 


Least  Conpatible 
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MOP  2b  QUESTIOSNAIHE/COMMEST  SHEET 


Participant  § 


You  are  requested  to  comment  on  your  assessment  of  the  graphics  displays 
you  have  seen  today.  Your  consents  should  include  vhat  you  liked/ disliked 
about  the  displays,  features  you  would  have  liked  the  displays  to  have, 
features  you  felt  were  superfluous  or  any  other  comment  you  feel  is  appli¬ 
cable.  Use  the  back  of  the  sheet  if  necessary. 


MO?  1b  500RS  SHEET 
{ SCENARIO  I) 


Participant  #  _____ 

1 .  Answer  Points 

a.  ______ 

b.  _ 

c .  ______ 

d.  _ 


2.  Answer  Points 


a. 

b. 

c. 

d. 


3.  Answer  Points 

a.  _ 

b.  _ 

c.  ______ 

d.  — . 


4.  Answer  Points 

a.  ______ 

b.  _ 

c . 

d.  _ 


5 •  Answer  Points 


a. 

b. 

c. 

d. 


6.  Answer  Points 

a.  _____ 

b.  _ 

c .  _ 

d. 


Participant's  Response 


Participant's  Response 


Participant's  Response 


Participant's  Response 


Participant's  Response 


Participant's  Response 


Total  Points  • 


TAB  11 

MOP  1b  SCOPE  SHEET 
( SCENARIO  II) 


Participant  #  _____ 

1.  Anavar  Points 


a* 

b. 

c. 

d. 


2.  Anavar  Points 


a. 

b. 

c. 

d. 


3 .  Anavar  Points 


a. 

b. 

c. 

d. 


a. 

b. 

c. 

d. 


4.  Anavar  Points 


5.  Anavar  Points 


a. 

b. 
e. 
d. 


6.  Anavar  Points 


a. 

b. 
e* 
d. 


Participant's  Response 


Participant's  Response 


Participant's  Response 


Participant's  Response 


Participant's  Response 


Participant's  Response 


Total  Points 


SOP  1b  DATA  CONSOLIDATION  FORM 


Display  Sige 


i 

I 

I 


Present 

ml— — .  T 


Map  Background 


Absent 

"V 
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MOP  2a  DATA  C0MS0LIDA7I0U  FCSM 


f .  Array 

J 

In  each  cell  enter  the  number  of  teat  participants  who  ranked  display  type 
j  above  display  type  i 


J  A  B  C  D 


TA3  15 

DATA  CONSOLIDATION  FORM  FOR  KENDALL  COEFFICIENT 


(4.)  Square  we h  individual  observation  and 
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(5)  5un  all  observations  ,  square  'tills  son  and  divide  by  the  total  aaber 
erf  observations*  i*e.» 
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Hethodoloar.  as  foil  ovine  sethodoloqj  sill  be  utilised  a  establish  a 
interval  seals  fros  tbs  ordinal  Judgneata  of  the  24  test  partlelpasta* 


(1)  2he  test  psTtlolpsats'  ordinal  rosponsee  vlll  be  allied  into  an  ft . 
assy  vbere  f44  is  tbs  aonber  of  participant*  vto  rank  display  1  above  display 
1*  14 
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(2)  A  p^ .  array  will  then  be  computed  where. 


On  the  diagonal  of  the  p  array,  we  set  p  -  .5 

*  J  i  J 

(3)  A  z  array  will  then  be  computed  where  z  is  the  standard  normal 

percentile  corresponding  to  p  .  The  z  array  will  contain  null  elements 
corresponding  to  any  ptJ  >  .98  or  p^  ^.02. 

(4)  If  the  z  array  has  no  null  elements  then  the  column  averager  will 
be  used  as  scale  Hlues  for  the  displays,  i.e.. 


(5)  If  the  z  •  array  has  null  elements  then  the  least  squares  method  will 
be  utilised,  i.e. ,J compute  column  averages  for  those  columns  which  are 
complete  and  use  these  averages  as  the  value  of  the  corresponding  display. 

For  incomplete  columns  write  a  set  of  linear  equations  of  the  form, 

ns  -  s  •  z 

where  denotes  the  set  of  n  elements  in  column  j  of  the  z  array.  Substi¬ 
tuting  for  the  ecale  values  already  determined,  solve  the  set  of  simultaneous 
equations  to  obtain  the  remaining  scale  values. 

Methodology  for  Calculating  Kendall  Coefficient  of  Concordance 

The  following  methodology  will  be  utilised  to  calculate  a  Kendall 
coefficient  of  concordance. 

(t)  The  teat  participants’  ordinal  rankings  of  the  various  displays  will 
be  tallied  into  an  r  array  such  that  if  the  ith  participant  ranked  the  jth 
display  as  the  "moat1tiompatible"  then  r  •  1.  If  the  ith  participant  ranked 
the  Jth  display  as  the  "second  most  compatible"  then  r  »  2,  etc. 

In  the  event  of  ties  the  observations  are  each  assigned  the  average  of  the 
ranks  they  would  have  been  aasigned  had  no  ties  occurred. 
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(2)  Sum  each  column  of  the  r  array,  i.e., 

24 


i-1 


(3)  Sum  the  Rs  and  divide  by  the  number  of  diaplaya,  i.e., 


8 


(4)  Calculate  the  sum  of  squared  deviations  of  each  R  from  IT,  i.e.. 


a 


R  ) 


2 


(5)  Calculate  the  Kendall  coefficient  which  is  the  ratio  of  a  to  the 
maximum  possible  sum  of  squared  deviations,  i.e., 

s 


W 


(6)  Convert  V  to  a  chi-sriare  statistic  with  3  degrees  of  freedom,  i.e., 

^  (3)  -  72V 

(7)  If  the  chi-square  statistic  is  greater  than  or  equal  to  6.25  then  the 
hypothesis  that  the  participants'  ratings  are  unrelated  can  be  rejected  at  a 
significance  level  of  •  .  1 . 
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